{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "automl-text-classification.ipynb",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0259a7ce8120"
      },
      "source": [
        "# Vertex AI: Create, Train, and Deploy an AutoML Text Classification Model\n",
        "\n",
        "## Learning Objective\n",
        "\n",
        "In this notebook, you learn how to:\n",
        "\n",
        "* Create a dataset and import data.\n",
        "* Train an AutoML model.\n",
        "* Get and review evaluations for the model.\n",
        "* Deploy a model to an endpoint.\n",
        "* Get online predictions.\n",
        "* Get batch predictions.\n",
        "\n",
        "## Introduction\n",
        "\n",
        "In this notebook, you use the \"Happy Moments\" sample dataset to train a model. The resulting model classifies happy moments into categories that reflect the causes of happiness. This notebook walks you through the major phases of building and using a text classification model on [Vertex AI](https://cloud.google.com/vertex-ai/docs/).\n",
        "\n",
        "Each learning objective will correspond to a __#TODO__ in this student lab notebook -- try to complete this notebook first and then review the [solution notebook](../solutions/automl_text_classification.ipynb).  \n",
        "\n",
        "**Make sure to enable the Vertex AI, Cloud Storage, and Compute Engine APIs.**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "db52a0a61fca"
      },
      "source": [
        "### Install additional packages\n",
        "\n",
        "This notebook uses the Python SDK for Vertex AI, which is contained in the `python-aiplatform` package. You must first install the package into your development environment."
      ]
    },
    {
     "cell_type": "code",
     "execution_count": 1,
     "metadata": {
      "id": "b75757581291"
     },
     "outputs": [
      {
       "name": "stdout",
       "output_type": "stream",
       "text": [
        "Requirement already satisfied: google-cloud-aiplatform in /opt/conda/lib/python3.7/site-packages (1.1.1)\n",
        "Collecting google-cloud-aiplatform\n",
        "  Downloading google_cloud_aiplatform-1.3.0-py2.py3-none-any.whl (1.3 MB)\n",
        "\u001b[K     |████████████████████████████████| 1.3 MB 8.6 MB/s eta 0:00:01\n",
        "\u001b[?25hRequirement already satisfied: google-cloud-storage in /opt/conda/lib/python3.7/site-packages (1.41.1)\n",
        "Collecting google-cloud-storage\n",
        "  Downloading google_cloud_storage-1.42.0-py2.py3-none-any.whl (105 kB)\n",
        "\u001b[K     |████████████████████████████████| 105 kB 48.2 MB/s eta 0:00:01\n",
        "\u001b[?25hCollecting jsonlines\n",
        "  Downloading jsonlines-2.0.0-py3-none-any.whl (6.3 kB)\n",
        "Requirement already satisfied: proto-plus>=1.10.1 in /opt/conda/lib/python3.7/site-packages (from google-cloud-aiplatform) (1.19.0)\n",
        "Requirement already satisfied: google-api-core[grpc]<3.0.0dev,>=1.26.0 in /opt/conda/lib/python3.7/site-packages (from google-cloud-aiplatform) (1.31.1)\n",
        "Requirement already satisfied: packaging>=14.3 in /opt/conda/lib/python3.7/site-packages (from google-cloud-aiplatform) (21.0)\n",
        "Requirement already satisfied: google-cloud-bigquery<3.0.0dev,>=1.15.0 in /opt/conda/lib/python3.7/site-packages (from google-cloud-aiplatform) (2.23.2)\n",
        "Requirement already satisfied: google-cloud-core<3.0dev,>=1.6.0 in /opt/conda/lib/python3.7/site-packages (from google-cloud-storage) (1.7.2)\n",
        "Requirement already satisfied: google-auth<3.0dev,>=1.25.0 in /opt/conda/lib/python3.7/site-packages (from google-cloud-storage) (1.34.0)\n",
        "Requirement already satisfied: google-resumable-media<3.0dev,>=1.3.0 in /opt/conda/lib/python3.7/site-packages (from google-cloud-storage) (1.3.2)\n",
        "Requirement already satisfied: requests<3.0.0dev,>=2.18.0 in /opt/conda/lib/python3.7/site-packages (from google-cloud-storage) (2.25.1)\n",
        "Requirement already satisfied: pytz in /opt/conda/lib/python3.7/site-packages (from google-api-core[grpc]<3.0.0dev,>=1.26.0->google-cloud-aiplatform) (2021.1)\n",
        "Requirement already satisfied: protobuf>=3.12.0 in /opt/conda/lib/python3.7/site-packages (from google-api-core[grpc]<3.0.0dev,>=1.26.0->google-cloud-aiplatform) (3.16.0)\n",
        "Requirement already satisfied: setuptools>=40.3.0 in /opt/conda/lib/python3.7/site-packages (from google-api-core[grpc]<3.0.0dev,>=1.26.0->google-cloud-aiplatform) (49.6.0.post20210108)\n",
        "Requirement already satisfied: six>=1.13.0 in /opt/conda/lib/python3.7/site-packages (from google-api-core[grpc]<3.0.0dev,>=1.26.0->google-cloud-aiplatform) (1.16.0)\n",
        "Requirement already satisfied: googleapis-common-protos<2.0dev,>=1.6.0 in /opt/conda/lib/python3.7/site-packages (from google-api-core[grpc]<3.0.0dev,>=1.26.0->google-cloud-aiplatform) (1.53.0)\n",
        "Requirement already satisfied: grpcio<2.0dev,>=1.29.0 in /opt/conda/lib/python3.7/site-packages (from google-api-core[grpc]<3.0.0dev,>=1.26.0->google-cloud-aiplatform) (1.38.1)\n",
        "Requirement already satisfied: rsa<5,>=3.1.4 in /opt/conda/lib/python3.7/site-packages (from google-auth<3.0dev,>=1.25.0->google-cloud-storage) (4.7.2)\n",
        "Requirement already satisfied: cachetools<5.0,>=2.0.0 in /opt/conda/lib/python3.7/site-packages (from google-auth<3.0dev,>=1.25.0->google-cloud-storage) (4.2.2)\n",
        "Requirement already satisfied: pyasn1-modules>=0.2.1 in /opt/conda/lib/python3.7/site-packages (from google-auth<3.0dev,>=1.25.0->google-cloud-storage) (0.2.7)\n",
        "Requirement already satisfied: google-crc32c<2.0dev,>=1.0 in /opt/conda/lib/python3.7/site-packages (from google-resumable-media<3.0dev,>=1.3.0->google-cloud-storage) (1.1.2)\n",
        "Requirement already satisfied: cffi>=1.0.0 in /opt/conda/lib/python3.7/site-packages (from google-crc32c<2.0dev,>=1.0->google-resumable-media<3.0dev,>=1.3.0->google-cloud-storage) (1.14.6)\n",
        "Requirement already satisfied: pycparser in /opt/conda/lib/python3.7/site-packages (from cffi>=1.0.0->google-crc32c<2.0dev,>=1.0->google-resumable-media<3.0dev,>=1.3.0->google-cloud-storage) (2.20)\n",
        "Requirement already satisfied: pyparsing>=2.0.2 in /opt/conda/lib/python3.7/site-packages (from packaging>=14.3->google-cloud-aiplatform) (2.4.7)\n",
        "Requirement already satisfied: pyasn1<0.5.0,>=0.4.6 in /opt/conda/lib/python3.7/site-packages (from pyasn1-modules>=0.2.1->google-auth<3.0dev,>=1.25.0->google-cloud-storage) (0.4.8)\n",
        "Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/lib/python3.7/site-packages (from requests<3.0.0dev,>=2.18.0->google-cloud-storage) (2021.5.30)\n",
        "Requirement already satisfied: chardet<5,>=3.0.2 in /opt/conda/lib/python3.7/site-packages (from requests<3.0.0dev,>=2.18.0->google-cloud-storage) (4.0.0)\n",
        "Requirement already satisfied: urllib3<1.27,>=1.21.1 in /opt/conda/lib/python3.7/site-packages (from requests<3.0.0dev,>=2.18.0->google-cloud-storage) (1.26.6)\n",
        "Requirement already satisfied: idna<3,>=2.5 in /opt/conda/lib/python3.7/site-packages (from requests<3.0.0dev,>=2.18.0->google-cloud-storage) (2.10)\n",
        "Installing collected packages: google-cloud-storage, jsonlines, google-cloud-aiplatform\n",
        "\u001b[33m  WARNING: The script tb-gcp-uploader is installed in '/home/jupyter/.local/bin' which is not on PATH.\n",
        "  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.\u001b[0m\n",
        "Successfully installed google-cloud-aiplatform-1.3.0 google-cloud-storage-1.42.0 jsonlines-2.0.0\n"
       ]
      }
     ],
     "source": [
      "# Setup your dependencies\n",
      "import os\n",
      "\n",
      "# The Google Cloud Notebook product has specific requirements\n",
      "IS_GOOGLE_CLOUD_NOTEBOOK = os.path.exists(\"/opt/deeplearning/metadata/env_version\")\n",
      "\n",
      "# Google Cloud Notebook requires dependencies to be installed with '--user'\n",
      "USER_FLAG = \"\"\n",
      "if IS_GOOGLE_CLOUD_NOTEBOOK:\n",
      "    USER_FLAG = \"--user\"\n",
      "\n",
      "# Upgrade the specified package to the newest available version\n",
      "! pip install {USER_FLAG} --upgrade google-cloud-aiplatform google-cloud-storage jsonlines"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
       "Please ignore any incompatibility warnings.\n"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "**Restart** the kernel before proceeding further (On the Notebook menu - Kernel - Restart Kernel).\n"
     ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WReHDGG5g0XY"
      },
      "source": [
        "### Set your project ID\n",
        "\n",
        "Finally, you must initialize the client library before you can send requests to the Vertex AI service. With the Python SDK, you initialize the client library as shown in the following cell. This tutorial also uses the Cloud Storage Python library for accessing batch prediction results.\n",
        "\n",
        "Be sure to provide the ID for your Google Cloud project in the `project` variable. This notebook uses the `us-central1` region, although you can change it to another region. \n",
        "\n",
        "**If you don't know your project ID**, you may be able to get your project ID using `gcloud`."
      ]
    },
    {
     "cell_type": "code",
     "execution_count": 1,
     "metadata": {
      "id": "oM1iC_MfAts1"
     },
     "outputs": [
      {
       "name": "stdout",
       "output_type": "stream",
       "text": [
        "Project ID:  qwiklabs-gcp-00-09d98f4803b0\n"
       ]
      }
     ],
     "source": [
      "# import necessary libraries\n",
      "import os\n",
      "from datetime import datetime\n",
      "\n",
      "import jsonlines\n",
      "from google.cloud import aiplatform, storage\n",
      "from google.protobuf import json_format\n",
      "\n",
      "PROJECT_ID = \"[your-project-id]\"\n",
      "REGION = \"us-central1\"\n",
      "\n",
      "# Get your Google Cloud project ID from gcloud\n",
      "if not os.getenv(\"IS_TESTING\"):\n",
      "    shell_output=!gcloud config list --format 'value(core.project)' 2>/dev/null\n",
      "    PROJECT_ID = shell_output[0]\n",
      "    print(\"Project ID: \", PROJECT_ID)\n",
      "\n",
      "aiplatform.init(project=PROJECT_ID, location=REGION)"
     ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "32c971919605"
      },
      "source": [
        "## Create a dataset and import your data\n",
        "\n",
        "The notebook uses the 'Happy Moments' dataset for demonstration purposes. You can change it to another text classification dataset that [conforms to the data preparation requirements](https://cloud.google.com/vertex-ai/docs/datasets/prepare-text#classification).\n",
        "\n",
        "Using the Python SDK, you can create a dataset and import the dataset in one call to `TextDataset.create()`, as shown in the following cell.\n",
        "\n",
        "Creating and importing data is a long-running operation. This next step can take a while. The sample waits for the operation to complete, outputting statements as the operation progresses. The statements contain the full name of the dataset that you will use in the following section.\n",
        "\n",
        "**Note**: You can close the noteboook while you wait for this operation to complete. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6caf82e5e84e"
      },
      "source": [
        "# TODO\n",
        "# Use a timestamp to ensure unique resources\n",
        "TIMESTAMP = # TODO: Your code goes here\n",
        "\n",
        "src_uris = \"gs://cloud-ml-data/NL-classification/happiness.csv\"\n",
        "display_name = f\"e2e-text-dataset-{TIMESTAMP}\""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
     "cell_type": "code",
     "execution_count": 4,
     "metadata": {
      "id": "d35b8b6b94ae"
     },
     "outputs": [
      {
       "name": "stdout",
       "output_type": "stream",
       "text": [
        "INFO:google.cloud.aiplatform.datasets.dataset:Creating TextDataset\n",
        "INFO:google.cloud.aiplatform.datasets.dataset:Create TextDataset backing LRO: projects/259224131669/locations/us-central1/datasets/7829200088927830016/operations/2215787784218607616\n",
        "INFO:google.cloud.aiplatform.datasets.dataset:TextDataset created. Resource name: projects/259224131669/locations/us-central1/datasets/7829200088927830016\n",
        "INFO:google.cloud.aiplatform.datasets.dataset:To use this TextDataset in another session:\n",
        "INFO:google.cloud.aiplatform.datasets.dataset:ds = aiplatform.TextDataset('projects/259224131669/locations/us-central1/datasets/7829200088927830016')\n",
        "INFO:google.cloud.aiplatform.datasets.dataset:Importing TextDataset data: projects/259224131669/locations/us-central1/datasets/7829200088927830016\n",
        "INFO:google.cloud.aiplatform.datasets.dataset:Import TextDataset data backing LRO: projects/259224131669/locations/us-central1/datasets/7829200088927830016/operations/7120207778425077760\n",
        "INFO:google.cloud.aiplatform.datasets.dataset:TextDataset data imported. Resource name: projects/259224131669/locations/us-central1/datasets/7829200088927830016\n"
       ]
      }
     ],
     "source": [
      "# TODO\n",
      "# create a dataset and import the dataset\n",
      "ds = # TODO: Your code goes here(\n",
      "    display_name=display_name,\n",
      "    gcs_source=src_uris,\n",
      "    import_schema_uri=aiplatform.schema.dataset.ioformat.text.single_label_classification,\n",
      "    sync=True,\n",
      ")"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "id": "5b3cc427353a"
     },
     "source": [
      "## Train your text classification model\n",
      "\n",
      "Once your dataset has finished importing data, you are ready to train your model. To do this, you first need the full resource name of your dataset, where the full name has the format `projects/[YOUR_PROJECT]/locations/us-central1/datasets/[YOUR_DATASET_ID]`. If you don't have the resource name handy, you can list all of the datasets in your project using `TextDataset.list()`. \n",
      "\n",
      "As shown in the following code block, you can pass in the display name of your dataset in the call to `list()` to filter the results.\n"
     ]
    },
    {
     "cell_type": "code",
     "execution_count": 5,
     "metadata": {
      "id": "52cf56f1c8a9"
     },
     "outputs": [
      {
       "name": "stdout",
       "output_type": "stream",
       "text": [
        "[<google.cloud.aiplatform.datasets.text_dataset.TextDataset object at 0x7fe2544d6dd0> \n",
        "resource name: projects/259224131669/locations/us-central1/datasets/7829200088927830016]\n"
       ]
      }
     ],
     "source": [
      "# TODO\n",
      "# list all of the datasets in your project\n",
      "datasets = # TODO: Your code goes here(filter=f'display_name=\"{display_name}\"')\n",
      "print(datasets)"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "id": "58df3e02df82"
     },
     "source": [
      "When you create a new model, you need a reference to the `TextDataset` object that corresponds to your dataset. You can use the `ds` variable you created previously when you created the dataset or you can also list all of your datasets to get a reference to your dataset. Each item returned from `TextDataset.list()` is an instance of `TextDataset`.\n",
      "\n",
      "The following code block shows how to instantiate a `TextDataset` object using a dataset ID. Note that this code is intentionally verbose for demonstration purposes."
     ]
    },
    {
     "cell_type": "code",
     "execution_count": 6,
     "metadata": {
      "id": "aa667203da03"
     },
     "outputs": [
      {
       "name": "stdout",
       "output_type": "stream",
       "text": [
        "Dataset ID: 7829200088927830016\n"
       ]
      }
     ],
     "source": [
      "# Get the dataset ID if it's not available\n",
      "dataset_id = \"[your-dataset-id]\"\n",
      "\n",
      "if dataset_id == \"[your-dataset-id]\":\n",
      "    # Use the reference to the new dataset captured when we created it\n",
      "    dataset_id = ds.resource_name.split(\"/\")[-1]\n",
      "    print(f\"Dataset ID: {dataset_id}\")\n",
      "\n",
      "text_dataset = aiplatform.TextDataset(dataset_id)"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "id": "68f10356cab9"
     },
     "source": [
      "Now you can begin training your model. Training the model is a two part process:\n",
      "\n",
      "1. **Define the training job.** You must provide a display name and the type of training you want when you define the training job.\n",
      "2. **Run the training job.** When you run the training job, you need to supply a reference to the dataset to use for training. At this step, you can also configure the data split percentages.\n",
      "\n",
      "You do not need to specify [data splits](https://cloud.google.com/vertex-ai/docs/general/ml-use). The training job has a default setting of  training 80%/ testing 10%/ validate 10% if you don't provide these values.\n",
      "\n",
      "To train your model, you call `AutoMLTextTrainingJob.run()` as shown in the following snippets. The method returns a reference to your new `Model` object.\n",
      "\n",
      "As with importing data into the dataset, training your model can take a substantial amount of time. The client library prints out operation status messages while the training pipeline operation processes. You must wait for the training process to complete before you can get the resource name and ID of your new model, which is required for model evaluation and model deployment.\n",
      "\n",
      "**Note**: You can close the notebook while you wait for the operation to complete."
     ]
    },
    {
     "cell_type": "code",
     "execution_count": 7,
     "metadata": {
      "id": "0aa0f01805ea"
     },
     "outputs": [],
     "source": [
      "# Define the training job\n",
      "training_job_display_name = f\"e2e-text-training-job-{TIMESTAMP}\"\n",
      "# TODO\n",
      "# constructs a AutoML Text Training Job\n",
      "job = # TODO: Your code goes here(\n",
      "    display_name=training_job_display_name,\n",
      "    prediction_type=\"classification\",\n",
      "    multi_label=False,\n",
      ")"
     ]
    },
    {
     "cell_type": "code",
     "execution_count": 8,
     "metadata": {
      "id": "1ec60baf2c51"
     },
     "outputs": [
      {
       "name": "stdout",
       "output_type": "stream",
       "text": [
        "INFO:google.cloud.aiplatform.training_jobs:View Training:\n",
        "https://console.cloud.google.com/ai/platform/locations/us-central1/training/1280924449289273344?project=259224131669\n",
        "INFO:google.cloud.aiplatform.training_jobs:AutoMLTextTrainingJob projects/259224131669/locations/us-central1/trainingPipelines/1280924449289273344 current state:\n",
        "PipelineState.PIPELINE_STATE_PENDING\n",
        "INFO:google.cloud.aiplatform.training_jobs:AutoMLTextTrainingJob projects/259224131669/locations/us-central1/trainingPipelines/1280924449289273344 current state:\n",
        "PipelineState.PIPELINE_STATE_PENDING\n",
        "INFO:google.cloud.aiplatform.training_jobs:AutoMLTextTrainingJob projects/259224131669/locations/us-central1/trainingPipelines/1280924449289273344 current state:\n",
        "PipelineState.PIPELINE_STATE_RUNNING\n",
        "INFO:google.cloud.aiplatform.training_jobs:AutoMLTextTrainingJob projects/259224131669/locations/us-central1/trainingPipelines/1280924449289273344 current state:\n",
        "PipelineState.PIPELINE_STATE_RUNNING\n",
        "INFO:google.cloud.aiplatform.training_jobs:AutoMLTextTrainingJob projects/259224131669/locations/us-central1/trainingPipelines/1280924449289273344 current state:\n",
        "PipelineState.PIPELINE_STATE_RUNNING\n",
        "INFO:google.cloud.aiplatform.training_jobs:AutoMLTextTrainingJob projects/259224131669/locations/us-central1/trainingPipelines/1280924449289273344 current state:\n",
        "PipelineState.PIPELINE_STATE_RUNNING\n",
        "INFO:google.cloud.aiplatform.training_jobs:AutoMLTextTrainingJob projects/259224131669/locations/us-central1/trainingPipelines/1280924449289273344 current state:\n",
        "PipelineState.PIPELINE_STATE_RUNNING\n",
        "INFO:google.cloud.aiplatform.training_jobs:AutoMLTextTrainingJob projects/259224131669/locations/us-central1/trainingPipelines/1280924449289273344 current state:\n",
        "PipelineState.PIPELINE_STATE_RUNNING\n",
        "INFO:google.cloud.aiplatform.training_jobs:AutoMLTextTrainingJob projects/259224131669/locations/us-central1/trainingPipelines/1280924449289273344 current state:\n",
        "PipelineState.PIPELINE_STATE_RUNNING\n",
        "INFO:google.cloud.aiplatform.training_jobs:AutoMLTextTrainingJob projects/259224131669/locations/us-central1/trainingPipelines/1280924449289273344 current state:\n",
        "PipelineState.PIPELINE_STATE_RUNNING\n",
        "INFO:google.cloud.aiplatform.training_jobs:AutoMLTextTrainingJob projects/259224131669/locations/us-central1/trainingPipelines/1280924449289273344 current state:\n",
        "PipelineState.PIPELINE_STATE_RUNNING\n",
        "INFO:google.cloud.aiplatform.training_jobs:AutoMLTextTrainingJob projects/259224131669/locations/us-central1/trainingPipelines/1280924449289273344 current state:\n",
        "PipelineState.PIPELINE_STATE_RUNNING\n",
        "INFO:google.cloud.aiplatform.training_jobs:AutoMLTextTrainingJob projects/259224131669/locations/us-central1/trainingPipelines/1280924449289273344 current state:\n",
        "PipelineState.PIPELINE_STATE_RUNNING\n",
        "INFO:google.cloud.aiplatform.training_jobs:AutoMLTextTrainingJob projects/259224131669/locations/us-central1/trainingPipelines/1280924449289273344 current state:\n",
        "PipelineState.PIPELINE_STATE_RUNNING\n",
        "INFO:google.cloud.aiplatform.training_jobs:AutoMLTextTrainingJob projects/259224131669/locations/us-central1/trainingPipelines/1280924449289273344 current state:\n",
        "PipelineState.PIPELINE_STATE_RUNNING\n",
        "INFO:google.cloud.aiplatform.training_jobs:AutoMLTextTrainingJob projects/259224131669/locations/us-central1/trainingPipelines/1280924449289273344 current state:\n",
        "PipelineState.PIPELINE_STATE_RUNNING\n",
        "INFO:google.cloud.aiplatform.training_jobs:AutoMLTextTrainingJob projects/259224131669/locations/us-central1/trainingPipelines/1280924449289273344 current state:\n",
        "PipelineState.PIPELINE_STATE_RUNNING\n",
        "INFO:google.cloud.aiplatform.training_jobs:AutoMLTextTrainingJob projects/259224131669/locations/us-central1/trainingPipelines/1280924449289273344 current state:\n",
        "PipelineState.PIPELINE_STATE_RUNNING\n",
        "INFO:google.cloud.aiplatform.training_jobs:AutoMLTextTrainingJob projects/259224131669/locations/us-central1/trainingPipelines/1280924449289273344 current state:\n",
        "PipelineState.PIPELINE_STATE_RUNNING\n",
        "INFO:google.cloud.aiplatform.training_jobs:AutoMLTextTrainingJob projects/259224131669/locations/us-central1/trainingPipelines/1280924449289273344 current state:\n",
        "PipelineState.PIPELINE_STATE_RUNNING\n",
        "INFO:google.cloud.aiplatform.training_jobs:AutoMLTextTrainingJob projects/259224131669/locations/us-central1/trainingPipelines/1280924449289273344 current state:\n",
        "PipelineState.PIPELINE_STATE_RUNNING\n",
        "INFO:google.cloud.aiplatform.training_jobs:AutoMLTextTrainingJob projects/259224131669/locations/us-central1/trainingPipelines/1280924449289273344 current state:\n",
        "PipelineState.PIPELINE_STATE_RUNNING\n",
        "INFO:google.cloud.aiplatform.training_jobs:AutoMLTextTrainingJob projects/259224131669/locations/us-central1/trainingPipelines/1280924449289273344 current state:\n",
        "PipelineState.PIPELINE_STATE_RUNNING\n",
        "INFO:google.cloud.aiplatform.training_jobs:AutoMLTextTrainingJob projects/259224131669/locations/us-central1/trainingPipelines/1280924449289273344 current state:\n",
        "PipelineState.PIPELINE_STATE_RUNNING\n",
        "INFO:google.cloud.aiplatform.training_jobs:AutoMLTextTrainingJob projects/259224131669/locations/us-central1/trainingPipelines/1280924449289273344 current state:\n",
        "PipelineState.PIPELINE_STATE_RUNNING\n",
        "INFO:google.cloud.aiplatform.training_jobs:AutoMLTextTrainingJob projects/259224131669/locations/us-central1/trainingPipelines/1280924449289273344 current state:\n",
        "PipelineState.PIPELINE_STATE_RUNNING\n",
        "INFO:google.cloud.aiplatform.training_jobs:AutoMLTextTrainingJob projects/259224131669/locations/us-central1/trainingPipelines/1280924449289273344 current state:\n",
        "PipelineState.PIPELINE_STATE_RUNNING\n",
        "INFO:google.cloud.aiplatform.training_jobs:AutoMLTextTrainingJob projects/259224131669/locations/us-central1/trainingPipelines/1280924449289273344 current state:\n",
        "PipelineState.PIPELINE_STATE_RUNNING\n",
        "INFO:google.cloud.aiplatform.training_jobs:AutoMLTextTrainingJob projects/259224131669/locations/us-central1/trainingPipelines/1280924449289273344 current state:\n",
        "PipelineState.PIPELINE_STATE_RUNNING\n",
        "INFO:google.cloud.aiplatform.training_jobs:AutoMLTextTrainingJob projects/259224131669/locations/us-central1/trainingPipelines/1280924449289273344 current state:\n",
        "PipelineState.PIPELINE_STATE_RUNNING\n",
        "INFO:google.cloud.aiplatform.training_jobs:AutoMLTextTrainingJob projects/259224131669/locations/us-central1/trainingPipelines/1280924449289273344 current state:\n",
        "PipelineState.PIPELINE_STATE_RUNNING\n",
        "INFO:google.cloud.aiplatform.training_jobs:AutoMLTextTrainingJob projects/259224131669/locations/us-central1/trainingPipelines/1280924449289273344 current state:\n",
        "PipelineState.PIPELINE_STATE_RUNNING\n",
        "INFO:google.cloud.aiplatform.training_jobs:AutoMLTextTrainingJob projects/259224131669/locations/us-central1/trainingPipelines/1280924449289273344 current state:\n",
        "PipelineState.PIPELINE_STATE_RUNNING\n",
        "INFO:google.cloud.aiplatform.training_jobs:AutoMLTextTrainingJob projects/259224131669/locations/us-central1/trainingPipelines/1280924449289273344 current state:\n",
        "PipelineState.PIPELINE_STATE_RUNNING\n",
        "INFO:google.cloud.aiplatform.training_jobs:AutoMLTextTrainingJob projects/259224131669/locations/us-central1/trainingPipelines/1280924449289273344 current state:\n",
        "PipelineState.PIPELINE_STATE_RUNNING\n",
        "INFO:google.cloud.aiplatform.training_jobs:AutoMLTextTrainingJob projects/259224131669/locations/us-central1/trainingPipelines/1280924449289273344 current state:\n",
        "PipelineState.PIPELINE_STATE_RUNNING\n",
        "INFO:google.cloud.aiplatform.training_jobs:AutoMLTextTrainingJob projects/259224131669/locations/us-central1/trainingPipelines/1280924449289273344 current state:\n",
        "PipelineState.PIPELINE_STATE_RUNNING\n",
        "INFO:google.cloud.aiplatform.training_jobs:AutoMLTextTrainingJob projects/259224131669/locations/us-central1/trainingPipelines/1280924449289273344 current state:\n",
        "PipelineState.PIPELINE_STATE_RUNNING\n",
        "INFO:google.cloud.aiplatform.training_jobs:AutoMLTextTrainingJob projects/259224131669/locations/us-central1/trainingPipelines/1280924449289273344 current state:\n",
        "PipelineState.PIPELINE_STATE_RUNNING\n",
        "INFO:google.cloud.aiplatform.training_jobs:AutoMLTextTrainingJob projects/259224131669/locations/us-central1/trainingPipelines/1280924449289273344 current state:\n",
        "PipelineState.PIPELINE_STATE_RUNNING\n",
        "INFO:google.cloud.aiplatform.training_jobs:AutoMLTextTrainingJob projects/259224131669/locations/us-central1/trainingPipelines/1280924449289273344 current state:\n",
        "PipelineState.PIPELINE_STATE_RUNNING\n",
        "INFO:google.cloud.aiplatform.training_jobs:AutoMLTextTrainingJob projects/259224131669/locations/us-central1/trainingPipelines/1280924449289273344 current state:\n",
        "PipelineState.PIPELINE_STATE_RUNNING\n",
        "INFO:google.cloud.aiplatform.training_jobs:AutoMLTextTrainingJob projects/259224131669/locations/us-central1/trainingPipelines/1280924449289273344 current state:\n",
        "PipelineState.PIPELINE_STATE_RUNNING\n",
        "INFO:google.cloud.aiplatform.training_jobs:AutoMLTextTrainingJob projects/259224131669/locations/us-central1/trainingPipelines/1280924449289273344 current state:\n",
        "PipelineState.PIPELINE_STATE_RUNNING\n",
        "INFO:google.cloud.aiplatform.training_jobs:AutoMLTextTrainingJob projects/259224131669/locations/us-central1/trainingPipelines/1280924449289273344 current state:\n",
        "PipelineState.PIPELINE_STATE_RUNNING\n",
        "INFO:google.cloud.aiplatform.training_jobs:AutoMLTextTrainingJob projects/259224131669/locations/us-central1/trainingPipelines/1280924449289273344 current state:\n",
        "PipelineState.PIPELINE_STATE_RUNNING\n",
        "INFO:google.cloud.aiplatform.training_jobs:AutoMLTextTrainingJob projects/259224131669/locations/us-central1/trainingPipelines/1280924449289273344 current state:\n",
        "PipelineState.PIPELINE_STATE_RUNNING\n",
        "INFO:google.cloud.aiplatform.training_jobs:AutoMLTextTrainingJob projects/259224131669/locations/us-central1/trainingPipelines/1280924449289273344 current state:\n",
        "PipelineState.PIPELINE_STATE_RUNNING\n",
        "INFO:google.cloud.aiplatform.training_jobs:AutoMLTextTrainingJob projects/259224131669/locations/us-central1/trainingPipelines/1280924449289273344 current state:\n",
        "PipelineState.PIPELINE_STATE_RUNNING\n",
        "INFO:google.cloud.aiplatform.training_jobs:AutoMLTextTrainingJob projects/259224131669/locations/us-central1/trainingPipelines/1280924449289273344 current state:\n",
        "PipelineState.PIPELINE_STATE_RUNNING\n",
        "INFO:google.cloud.aiplatform.training_jobs:AutoMLTextTrainingJob projects/259224131669/locations/us-central1/trainingPipelines/1280924449289273344 current state:\n",
        "PipelineState.PIPELINE_STATE_RUNNING\n",
        "INFO:google.cloud.aiplatform.training_jobs:AutoMLTextTrainingJob projects/259224131669/locations/us-central1/trainingPipelines/1280924449289273344 current state:\n",
        "PipelineState.PIPELINE_STATE_RUNNING\n",
        "INFO:google.cloud.aiplatform.training_jobs:AutoMLTextTrainingJob projects/259224131669/locations/us-central1/trainingPipelines/1280924449289273344 current state:\n",
        "PipelineState.PIPELINE_STATE_RUNNING\n",
        "INFO:google.cloud.aiplatform.training_jobs:AutoMLTextTrainingJob projects/259224131669/locations/us-central1/trainingPipelines/1280924449289273344 current state:\n",
        "PipelineState.PIPELINE_STATE_RUNNING\n",
        "INFO:google.cloud.aiplatform.training_jobs:AutoMLTextTrainingJob projects/259224131669/locations/us-central1/trainingPipelines/1280924449289273344 current state:\n",
        "PipelineState.PIPELINE_STATE_RUNNING\n",
        "INFO:google.cloud.aiplatform.training_jobs:AutoMLTextTrainingJob projects/259224131669/locations/us-central1/trainingPipelines/1280924449289273344 current state:\n",
        "PipelineState.PIPELINE_STATE_RUNNING\n",
        "INFO:google.cloud.aiplatform.training_jobs:AutoMLTextTrainingJob run completed. Resource name: projects/259224131669/locations/us-central1/trainingPipelines/1280924449289273344\n",
        "INFO:google.cloud.aiplatform.training_jobs:Model available at projects/259224131669/locations/us-central1/models/274218199967334400\n"
       ]
      }
     ],
     "source": [
      "model_display_name = f\"e2e-text-classification-model-{TIMESTAMP}\"\n",
      "\n",
      "# TODO\n",
      "# Run the training job\n",
      "model = # TODO: Your code goes here(\n",
      "    dataset=text_dataset,\n",
      "    model_display_name=model_display_name,\n",
      "    training_fraction_split=0.7,\n",
      "    validation_fraction_split=0.2,\n",
      "    test_fraction_split=0.1,\n",
      "    sync=True,\n",
      ")"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "id": "caaa3f32b12e"
     },
     "source": [
      "## Get and review model evaluation scores\n",
      "\n",
      "After your model has finished training, you can review the evaluation scores for it.\n",
      "\n",
      "First, you need to get a reference to the new model.  As with datasets, you can either use the reference to the `model` variable you created when deployed the model or you can list all of the models in your project. When listing your models, you can provide filter criteria to narrow down your search."
     ]
    },
    {
     "cell_type": "code",
     "execution_count": 9,
     "metadata": {
      "id": "b0bb6be8621a"
     },
     "outputs": [
      {
       "name": "stdout",
       "output_type": "stream",
       "text": [
        "[<google.cloud.aiplatform.models.Model object at 0x7fe24dedda90> \n",
        "resource name: projects/259224131669/locations/us-central1/models/274218199967334400]\n"
       ]
      }
     ],
     "source": [
      "# TODO\n",
      "# list the aiplatform model\n",
      "models = # TODO: Your code goes here(filter=f'display_name=\"{model_display_name}\"')\n",
      "print(models)"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "id": "8481b6878ed2"
     },
     "source": [
      "Using the model name (in the format `projects/[PROJECT_NAME]/locations/us-central1/models/[MODEL_ID]`), you can get its model evaluations. To get model evaluations, you must use the underlying service client.\n",
      "\n",
      "Building a service client requires that you provide the name of the regionalized hostname used for your model. In this tutorial, the hostname is `us-central1-aiplatform.googleapis.com` because the model was created in the `us-central1` location."
     ]
    },
    {
     "cell_type": "code",
     "execution_count": 10,
     "metadata": {
      "id": "a8443fc8861f"
     },
     "outputs": [
      {
       "name": "stdout",
       "output_type": "stream",
       "text": [
        "Model name: projects/259224131669/locations/us-central1/models/274218199967334400\n"
       ]
      }
     ],
     "source": [
      "# Get the ID of the model\n",
      "model_name = \"[your-model-resource-name]\"\n",
      "if model_name == \"[your-model-resource-name]\":\n",
      "    # Use the `resource_name` of the Model instance you created previously\n",
      "    model_name = model.resource_name\n",
      "    print(f\"Model name: {model_name}\")\n",
      "\n",
      "\n",
      "# Get a reference to the Model Service client\n",
      "client_options = {\"api_endpoint\": \"us-central1-aiplatform.googleapis.com\"}\n",
      "model_service_client = aiplatform.gapic.ModelServiceClient(\n",
      "    client_options=client_options\n",
      ")"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "id": "b8a788593609"
     },
     "source": [
      "Before you can view the model evaluation you must first list all of the evaluations for that model. Each model can have multiple evaluations, although a new model is likely to only have one. "
     ]
    },
    {
     "cell_type": "code",
     "execution_count": 11,
     "metadata": {
      "id": "fdcb045e29f2"
     },
     "outputs": [],
     "source": [
      "model_evaluations = model_service_client.list_model_evaluations(parent=model_name)\n",
      "model_evaluation = list(model_evaluations)[0]"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "id": "cd7d3afae05c"
     },
     "source": [
      "Now that you have the model evaluation, you can look at your model's scores. If you have questions about what the scores mean, review the [public documentation](https://cloud.google.com/vertex-ai/docs/training/evaluating-automl-models#text).\n",
      "\n",
      "The results returned from the service are formatted as [`google.protobuf.Value`](https://googleapis.dev/python/protobuf/latest/google/protobuf/struct_pb2.html) objects. You can transform the return object as a `dict` for easier reading and parsing."
     ]
    },
    {
     "cell_type": "code",
     "execution_count": 12,
     "metadata": {
      "id": "6eb9ccb0a0a0"
     },
     "outputs": [
      {
       "name": "stdout",
       "output_type": "stream",
       "text": [
        "Area under precision-recall curve (AuPRC): 0.9556533\n",
        "\n",
        "\n",
        "\trecallAt1: 0.8852321\n",
        "\tprecisionAt1: 0.8852321\n",
        "\trecall: 1.0\n",
        "\tprecision: 0.14285715\n",
        "\tf1ScoreAt1: 0.8852321\n",
        "\tf1Score: 0.25\n",
        "\n",
        "\n",
        "\tf1Score: 0.82728595\n",
        "\tf1ScoreAt1: 0.8852321\n",
        "\trecall: 0.96202534\n",
        "\tprecision: 0.72565246\n",
        "\trecallAt1: 0.8852321\n",
        "\tprecisionAt1: 0.8852321\n",
        "\tconfidenceThreshold: 0.05\n",
        "\n",
        "\n",
        "\tprecision: 0.7781617\n",
        "\tprecisionAt1: 0.8852321\n",
        "\tf1Score: 0.8556231\n",
        "\tf1ScoreAt1: 0.8852321\n",
        "\trecallAt1: 0.8852321\n",
        "\trecall: 0.950211\n",
        "\tconfidenceThreshold: 0.1\n",
        "\n",
        "\n",
        "\tf1ScoreAt1: 0.8852321\n",
        "\tprecisionAt1: 0.8852321\n",
        "\trecallAt1: 0.8852321\n",
        "\trecall: 0.9409283\n",
        "\tf1Score: 0.87075365\n",
        "\tconfidenceThreshold: 0.15\n",
        "\tprecision: 0.8103198\n",
        "\n",
        "\n",
        "\trecallAt1: 0.8852321\n",
        "\trecall: 0.9316456\n",
        "\tf1Score: 0.87653834\n",
        "\tprecisionAt1: 0.8852321\n",
        "\tprecision: 0.82758623\n",
        "\tconfidenceThreshold: 0.2\n",
        "\tf1ScoreAt1: 0.8852321\n",
        "\n",
        "\n",
        "\tf1ScoreAt1: 0.8852321\n",
        "\trecall: 0.9206751\n",
        "\tf1Score: 0.8794841\n",
        "\trecallAt1: 0.8852321\n",
        "\tprecisionAt1: 0.8852321\n",
        "\tconfidenceThreshold: 0.25\n",
        "\tprecision: 0.841821\n",
        "\n",
        "\n",
        "\tprecisionAt1: 0.8852321\n",
        "\trecall: 0.9105485\n",
        "\tconfidenceThreshold: 0.3\n",
        "\tprecision: 0.85363925\n",
        "\trecallAt1: 0.8852321\n",
        "\tf1ScoreAt1: 0.8852321\n",
        "\tf1Score: 0.881176\n",
        "\n",
        "\n",
        "\tf1Score: 0.8820132\n",
        "\tprecisionAt1: 0.8852321\n",
        "\tprecision: 0.86279255\n",
        "\trecallAt1: 0.8852321\n",
        "\trecall: 0.9021097\n",
        "\tf1ScoreAt1: 0.8852321\n",
        "\tconfidenceThreshold: 0.35\n",
        "\n",
        "\n",
        "\tconfidenceThreshold: 0.4\n",
        "\tf1ScoreAt1: 0.8854123\n",
        "\trecall: 0.8953586\n",
        "\tf1Score: 0.8871237\n",
        "\tprecisionAt1: 0.88728815\n",
        "\tprecision: 0.87903893\n",
        "\trecallAt1: 0.8835443\n",
        "\n",
        "\n",
        "\trecall: 0.8835443\n",
        "\tprecisionAt1: 0.8906917\n",
        "\tf1Score: 0.8861617\n",
        "\trecallAt1: 0.8801688\n",
        "\tconfidenceThreshold: 0.45\n",
        "\tf1ScoreAt1: 0.885399\n",
        "\tprecision: 0.88879454\n",
        "\n",
        "\n",
        "\trecallAt1: 0.87257385\n",
        "\tprecisionAt1: 0.8913793\n",
        "\tconfidenceThreshold: 0.5\n",
        "\tf1Score: 0.88187635\n",
        "\trecall: 0.87257385\n",
        "\tprecision: 0.8913793\n",
        "\tf1ScoreAt1: 0.88187635\n",
        "\n",
        "\n",
        "\tf1Score: 0.88093185\n",
        "\tprecisionAt1: 0.9011474\n",
        "\trecall: 0.8616034\n",
        "\tf1ScoreAt1: 0.88093185\n",
        "\tprecision: 0.9011474\n",
        "\trecallAt1: 0.8616034\n",
        "\tconfidenceThreshold: 0.55\n",
        "\n",
        "\n",
        "\tconfidenceThreshold: 0.6\n",
        "\trecall: 0.85316455\n",
        "\trecallAt1: 0.85316455\n",
        "\tprecisionAt1: 0.9124549\n",
        "\tprecision: 0.9124549\n",
        "\tf1Score: 0.88181424\n",
        "\tf1ScoreAt1: 0.88181424\n",
        "\n",
        "\n",
        "\tconfidenceThreshold: 0.65\n",
        "\trecall: 0.8413502\n",
        "\tf1ScoreAt1: 0.8784141\n",
        "\tf1Score: 0.8784141\n",
        "\tprecisionAt1: 0.918894\n",
        "\trecallAt1: 0.8413502\n",
        "\tprecision: 0.918894\n",
        "\n",
        "\n",
        "\trecall: 0.8303797\n",
        "\tprecision: 0.92742693\n",
        "\tf1ScoreAt1: 0.8762244\n",
        "\tf1Score: 0.8762244\n",
        "\trecallAt1: 0.8303797\n",
        "\tconfidenceThreshold: 0.7\n",
        "\tprecisionAt1: 0.92742693\n",
        "\n",
        "\n",
        "\tprecisionAt1: 0.9354528\n",
        "\trecallAt1: 0.8194093\n",
        "\tconfidenceThreshold: 0.75\n",
        "\tprecision: 0.9354528\n",
        "\tf1Score: 0.8735943\n",
        "\tf1ScoreAt1: 0.8735943\n",
        "\trecall: 0.8194093\n",
        "\n",
        "\n",
        "\tprecisionAt1: 0.94077\n",
        "\tprecision: 0.94077\n",
        "\trecall: 0.8042194\n",
        "\tconfidenceThreshold: 0.8\n",
        "\tf1ScoreAt1: 0.867152\n",
        "\trecallAt1: 0.8042194\n",
        "\tf1Score: 0.867152\n",
        "\n",
        "\n",
        "\tprecisionAt1: 0.9483806\n",
        "\trecallAt1: 0.7907173\n",
        "\trecall: 0.7907173\n",
        "\tprecision: 0.9483806\n",
        "\tf1ScoreAt1: 0.8624022\n",
        "\tconfidenceThreshold: 0.85\n",
        "\tf1Score: 0.8624022\n",
        "\n",
        "\n",
        "\trecallAt1: 0.7763713\n",
        "\tprecisionAt1: 0.9533679\n",
        "\tf1Score: 0.855814\n",
        "\trecall: 0.7763713\n",
        "\tf1ScoreAt1: 0.855814\n",
        "\tprecision: 0.9533679\n",
        "\tconfidenceThreshold: 0.875\n",
        "\n",
        "\n",
        "\tconfidenceThreshold: 0.9\n",
        "\tf1ScoreAt1: 0.8509638\n",
        "\tprecision: 0.96072185\n",
        "\trecallAt1: 0.76371306\n",
        "\trecall: 0.76371306\n",
        "\tprecisionAt1: 0.96072185\n",
        "\tf1Score: 0.8509638\n",
        "\n",
        "\n",
        "\tf1ScoreAt1: 0.84578997\n",
        "\tconfidenceThreshold: 0.91\n",
        "\tprecision: 0.9623251\n",
        "\tprecisionAt1: 0.9623251\n",
        "\tf1Score: 0.84578997\n",
        "\trecall: 0.75443035\n",
        "\trecallAt1: 0.75443035\n",
        "\n",
        "\n",
        "\tconfidenceThreshold: 0.92\n",
        "\trecall: 0.7443038\n",
        "\trecallAt1: 0.7443038\n",
        "\tprecisionAt1: 0.9639344\n",
        "\tf1ScoreAt1: 0.84000003\n",
        "\tprecision: 0.9639344\n",
        "\tf1Score: 0.84000003\n",
        "\n",
        "\n",
        "\trecallAt1: 0.735865\n",
        "\tf1ScoreAt1: 0.83524907\n",
        "\tf1Score: 0.83524907\n",
        "\tprecisionAt1: 0.96567\n",
        "\tconfidenceThreshold: 0.93\n",
        "\tprecision: 0.96567\n",
        "\trecall: 0.735865\n",
        "\n",
        "\n",
        "\tf1ScoreAt1: 0.8294798\n",
        "\trecall: 0.7265823\n",
        "\trecallAt1: 0.7265823\n",
        "\tf1Score: 0.8294798\n",
        "\tprecision: 0.96633\n",
        "\tconfidenceThreshold: 0.94\n",
        "\tprecisionAt1: 0.96633\n",
        "\n",
        "\n",
        "\tf1Score: 0.8214112\n",
        "\trecallAt1: 0.7122363\n",
        "\tf1ScoreAt1: 0.8214112\n",
        "\trecall: 0.7122363\n",
        "\tprecision: 0.97011495\n",
        "\tconfidenceThreshold: 0.95\n",
        "\tprecisionAt1: 0.97011495\n",
        "\n",
        "\n",
        "\tprecision: 0.97402596\n",
        "\tf1ScoreAt1: 0.81200784\n",
        "\tconfidenceThreshold: 0.96\n",
        "\trecallAt1: 0.6962025\n",
        "\tf1Score: 0.81200784\n",
        "\trecall: 0.6962025\n",
        "\tprecisionAt1: 0.97402596\n",
        "\n",
        "\n",
        "\trecallAt1: 0.68016875\n",
        "\tf1Score: 0.8023892\n",
        "\tprecisionAt1: 0.9781553\n",
        "\trecall: 0.68016875\n",
        "\tprecision: 0.9781553\n",
        "\tf1ScoreAt1: 0.8023892\n",
        "\tconfidenceThreshold: 0.97\n",
        "\n",
        "\n",
        "\trecall: 0.64978904\n",
        "\tprecision: 0.98214287\n",
        "\tprecisionAt1: 0.98214287\n",
        "\tconfidenceThreshold: 0.98\n",
        "\trecallAt1: 0.64978904\n",
        "\tf1ScoreAt1: 0.7821229\n",
        "\tf1Score: 0.7821229\n",
        "\n",
        "\n",
        "\trecall: 0.5915612\n",
        "\tf1Score: 0.74062335\n",
        "\tprecisionAt1: 0.990113\n",
        "\tprecision: 0.990113\n",
        "\trecallAt1: 0.5915612\n",
        "\tconfidenceThreshold: 0.99\n",
        "\tf1ScoreAt1: 0.74062335\n",
        "\n",
        "\n",
        "\tprecision: 0.9923195\n",
        "\tprecisionAt1: 0.9923195\n",
        "\trecallAt1: 0.54514766\n",
        "\tf1ScoreAt1: 0.7037037\n",
        "\tf1Score: 0.7037037\n",
        "\trecall: 0.54514766\n",
        "\tconfidenceThreshold: 0.995\n",
        "\n",
        "\n",
        "\tprecisionAt1: 0.99206346\n",
        "\tprecision: 0.99206346\n",
        "\trecallAt1: 0.5274262\n",
        "\tconfidenceThreshold: 0.996\n",
        "\trecall: 0.5274262\n",
        "\tf1ScoreAt1: 0.68870527\n",
        "\tf1Score: 0.68870527\n",
        "\n",
        "\n",
        "\tprecision: 0.9933665\n",
        "\tprecisionAt1: 0.9933665\n",
        "\tf1Score: 0.67002237\n",
        "\trecall: 0.50548524\n",
        "\tf1ScoreAt1: 0.67002237\n",
        "\tconfidenceThreshold: 0.997\n",
        "\trecallAt1: 0.50548524\n",
        "\n",
        "\n",
        "\tconfidenceThreshold: 0.998\n",
        "\tprecision: 0.9929453\n",
        "\trecallAt1: 0.4751055\n",
        "\tprecisionAt1: 0.9929453\n",
        "\trecall: 0.4751055\n",
        "\tf1Score: 0.64269406\n",
        "\tf1ScoreAt1: 0.64269406\n",
        "\n",
        "\n",
        "\tf1ScoreAt1: 0.58064514\n",
        "\tconfidenceThreshold: 0.999\n",
        "\trecall: 0.4101266\n",
        "\tprecision: 0.993865\n",
        "\tprecisionAt1: 0.993865\n",
        "\trecallAt1: 0.4101266\n",
        "\tf1Score: 0.58064514\n",
        "\n",
        "\n",
        "\tconfidenceThreshold: 1.0\n",
        "\tprecisionAt1: 1.0\n",
        "\tf1Score: 0.023352792\n",
        "\trecallAt1: 0.011814346\n",
        "\tprecision: 1.0\n",
        "\trecall: 0.011814346\n",
        "\tf1ScoreAt1: 0.023352792\n"
       ]
      }
     ],
     "source": [
      "model_eval_dict = json_format.MessageToDict(model_evaluation._pb)\n",
      "metrics = model_eval_dict[\"metrics\"]\n",
      "confidence_metrics = metrics[\"confidenceMetrics\"]\n",
      "\n",
      "print(f'Area under precision-recall curve (AuPRC): {metrics[\"auPrc\"]}')\n",
      "for confidence_scores in confidence_metrics:\n",
      "    metrics = confidence_scores.keys()\n",
      "    print(\"\\n\")\n",
      "    for metric in metrics:\n",
      "        print(f\"\\t{metric}: {confidence_scores[metric]}\")"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "id": "b5dbe4dbaa60"
     },
     "source": [
      "## Deploy your text classification model\n",
      "\n",
      "Once your model has completed training, you must deploy it to an _endpoint_ to get online predictions from it. When you deploy the model to an endpoint, a copy of the model is made on the endpoint with a new resource name and display name.\n",
      "\n",
      "You can deploy multiple models to the same endpoint and split traffic between the various models assigned to the endpoint. However, you must deploy one model at a time to the endpoint. To change the traffic split percentages, you must assign new values on your second (and subsequent) models each time you deploy a new model.\n",
      "\n",
      "The following code block demonstrates how to deploy a model. The code snippet relies on the Python SDK to create a new endpoint for deployment. The call to `model.deploy()` returns a reference to an `Endpoint` object--you need this reference for online predictions in the next section."
     ]
    },
    {
     "cell_type": "code",
     "execution_count": 13,
     "metadata": {
      "id": "19bc4a55ccfe"
     },
     "outputs": [
      {
       "name": "stdout",
       "output_type": "stream",
       "text": [
        "INFO:google.cloud.aiplatform.models:Creating Endpoint\n",
        "INFO:google.cloud.aiplatform.models:Create Endpoint backing LRO: projects/259224131669/locations/us-central1/endpoints/7980783159979540480/operations/3267096822232907776\n",
        "INFO:google.cloud.aiplatform.models:Endpoint created. Resource name: projects/259224131669/locations/us-central1/endpoints/7980783159979540480\n",
        "INFO:google.cloud.aiplatform.models:To use this Endpoint in another session:\n",
        "INFO:google.cloud.aiplatform.models:endpoint = aiplatform.Endpoint('projects/259224131669/locations/us-central1/endpoints/7980783159979540480')\n",
        "INFO:google.cloud.aiplatform.models:Deploying model to Endpoint : projects/259224131669/locations/us-central1/endpoints/7980783159979540480\n",
        "INFO:google.cloud.aiplatform.models:Deploy Endpoint model backing LRO: projects/259224131669/locations/us-central1/endpoints/7980783159979540480/operations/7878782840660295680\n",
        "INFO:google.cloud.aiplatform.models:Endpoint model deployed. Resource name: projects/259224131669/locations/us-central1/endpoints/7980783159979540480\n"
       ]
      }
     ],
     "source": [
      "deployed_model_display_name = f\"e2e-deployed-text-classification-model-{TIMESTAMP}\"\n",
      "\n",
      "# TODO\n",
      "# deploy a model\n",
      "endpoint = # TODO: Your code goes here(\n",
      "    deployed_model_display_name=deployed_model_display_name, sync=True\n",
      ")"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "id": "531da446035b"
     },
     "source": [
      "In case you didn't record the name of the new endpoint, you can get a list of all your endpoints as you did before with datasets and models. For each endpoint, you can list the models deployed to that endpoint. To get a reference to the model that you just deployed, you can check the `display_name` of each model deployed to the endpoint against the model you're looking for."
     ]
    },
    {
     "cell_type": "code",
     "execution_count": 14,
     "metadata": {
      "id": "f61fb44181b4"
     },
     "outputs": [
      {
       "name": "stdout",
       "output_type": "stream",
       "text": [
        "[<google.cloud.aiplatform.models.Endpoint object at 0x7fe24de99d10> \n",
        "resource name: projects/259224131669/locations/us-central1/endpoints/7980783159979540480]\n"
       ]
      }
     ],
     "source": [
      "endpoints = aiplatform.Endpoint.list()\n",
      "\n",
      "endpoint_with_deployed_model = []\n",
      "\n",
      "for endpoint_ in endpoints:\n",
      "    for model in endpoint_.list_models():\n",
      "        if model.display_name.find(deployed_model_display_name) == 0:\n",
      "            endpoint_with_deployed_model.append(endpoint_)\n",
      "\n",
      "print(endpoint_with_deployed_model)"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "id": "351a6e8be3a5"
     },
     "source": [
      "## Get online predictions from your model\n",
      "\n",
      "Now that you have your endpoint's resource name, you can get online predictions from the text classification model. To get the online prediction, you send a prediction request to your endpoint."
     ]
    },
    {
     "cell_type": "code",
     "execution_count": 15,
     "metadata": {
      "id": "953b333fc0fc"
     },
     "outputs": [
      {
       "name": "stdout",
       "output_type": "stream",
       "text": [
        "Endpoint name: projects/259224131669/locations/us-central1/endpoints/7980783159979540480\n",
        "Prediction ID: 5078374828348538880\n",
        "Prediction display name: affection\n",
        "Prediction confidence score: 4.789760350831784e-05\n",
        "Prediction ID: 4213683699893403648\n",
        "Prediction display name: achievement\n",
        "Prediction confidence score: 0.9997887015342712\n",
        "Prediction ID: 7384217837562232832\n",
        "Prediction display name: enjoy_the_moment\n",
        "Prediction confidence score: 5.908047751290724e-05\n",
        "Prediction ID: 466688809921150976\n",
        "Prediction display name: bonding\n",
        "Prediction confidence score: 2.292021417815704e-05\n",
        "Prediction ID: 1619610314527997952\n",
        "Prediction display name: leisure\n",
        "Prediction confidence score: 5.406829222920351e-05\n",
        "Prediction ID: 8825369718320791552\n",
        "Prediction display name: nature\n",
        "Prediction confidence score: 2.831711753970012e-06\n",
        "Prediction ID: 2772531819134844928\n",
        "Prediction display name: exercise\n",
        "Prediction confidence score: 2.4389159079873934e-05\n"
       ]
      }
     ],
     "source": [
      "endpoint_name = \"[your-endpoint-name]\"\n",
      "if endpoint_name == \"[your-endpoint-name]\":\n",
      "    endpoint_name = endpoint.resource_name\n",
      "\n",
      "print(f\"Endpoint name: {endpoint_name}\")\n",
      "\n",
      "endpoint = aiplatform.Endpoint(endpoint_name)\n",
      "content = \"I got a high score on my math final!\"\n",
      "\n",
      "# TODO\n",
      "# send a prediction request to your endpoint\n",
      "response = # TODO: Your code goes here(instances=[{\"content\": content}])\n",
      "\n",
      "for prediction_ in response.predictions:\n",
      "    ids = prediction_[\"ids\"]\n",
      "    display_names = prediction_[\"displayNames\"]\n",
      "    confidence_scores = prediction_[\"confidences\"]\n",
      "    for count, id in enumerate(ids):\n",
      "        print(f\"Prediction ID: {id}\")\n",
      "        print(f\"Prediction display name: {display_names[count]}\")\n",
      "        print(f\"Prediction confidence score: {confidence_scores[count]}\")"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "id": "f18811cd0477"
     },
     "source": [
      "## Get batch predictions from your model\n",
      "\n",
      "You can get batch predictions from a text classification model without deploying it. You must first format all of your prediction instances (prediction input) in JSONL format and you must store the JSONL file in a Google Cloud Storage bucket. You must also provide a Google Cloud Storage bucket to hold your prediction output.\n",
      "\n",
      "To start, you must first create your predictions input file in JSONL format. Each line in the JSONL document needs to be formatted like so:\n",
      "\n",
      "```\n",
      "{ \"content\": \"gs://sourcebucket/datasets/texts/source_text.txt\", \"mimeType\": \"text/plain\"}\n",
      "```\n",
      "\n",
      "The `content` field in the JSON structure must be a Google Cloud Storage URI to another document that contains the text input for prediction.\n",
      "[See the documentation for more information.](https://cloud.google.com/ai-platform-unified/docs/predictions/batch-predictions#text)"
     ]
    },
    {
     "cell_type": "code",
     "execution_count": 16,
     "metadata": {
      "id": "e4b838cbcd99"
     },
     "outputs": [],
     "source": [
      "instances = [\n",
      "    \"We hiked through the woods and up the hill to the ice caves\",\n",
      "    \"My kitten is so cute\",\n",
      "]\n",
      "input_file_name = \"batch-prediction-input.jsonl\""
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "id": "76ac422ab8dd"
     },
     "source": [
      "For batch prediction, you must supply the following:\n",
      "\n",
      "+ All of your prediction instances as individual files on Google Cloud Storage, as TXT files for your instances\n",
      "+ A JSONL file that lists the URIs of all your prediction instances\n",
      "+ A Google Cloud Storage bucket to hold the output from batch prediction\n",
      "\n",
      "For this tutorial, the following cells create a new Storage bucket, upload individual prediction instances as text files to the bucket, and then create the JSONL file with the URIs of your prediction instances."
     ]
    },
    {
     "cell_type": "code",
     "execution_count": 17,
     "metadata": {
      "id": "1e0759fbb219"
     },
     "outputs": [
      {
       "name": "stdout",
       "output_type": "stream",
       "text": [
        "Creating gs://qwiklabs-gcp-00-09d98f4803b0/...\n"
       ]
      }
     ],
     "source": [
      "TIMESTAMP = datetime.now().strftime(\"%Y%m%d%H%M%S\")\n",
      "BUCKET_NAME = \"[your-bucket-name]\"\n",
      "\n",
      "if BUCKET_NAME == \"\" or BUCKET_NAME is None or BUCKET_NAME == \"[your-bucket-name]\":\n",
      "    BUCKET_NAME = f\"automl-text-notebook-{TIMESTAMP}\"\n",
      "\n",
      "BUCKET_URI = f\"gs://{BUCKET_NAME}\"\n",
      "\n",
      "! gcloud storage buckets create --location $REGION $BUCKET_URI"     ]
    },
    {
     "cell_type": "code",
     "execution_count": 18,
     "metadata": {
      "id": "8b7cabbb86ad"
     },
     "outputs": [],
     "source": [
      "# Instantiate the Storage client and create the new bucket\n",
      "storage = storage.Client()\n",
      "bucket = storage.bucket(BUCKET_NAME)\n",
      "\n",
      "# Iterate over the prediction instances, creating a new TXT file\n",
      "# for each.\n",
      "input_file_data = []\n",
      "for count, instance in enumerate(instances):\n",
      "    instance_name = f\"input_{count}.txt\"\n",
      "    instance_file_uri = f\"{BUCKET_URI}/{instance_name}\"\n",
      "\n",
      "    # Add the data to store in the JSONL input file.\n",
      "    tmp_data = {\"content\": instance_file_uri, \"mimeType\": \"text/plain\"}\n",
      "    input_file_data.append(tmp_data)\n",
      "\n",
      "    # Create the new instance file\n",
      "    blob = bucket.blob(instance_name)\n",
      "    blob.upload_from_string(instance)\n",
      "\n",
      "input_str = \"\\n\".join([str(d) for d in input_file_data])\n",
      "file_blob = bucket.blob(f\"{input_file_name}\")\n",
      "file_blob.upload_from_string(input_str)"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "id": "31c262320610"
     },
     "source": [
      "Now that you have the bucket with the prediction instances ready, you can send a batch prediction request to Vertex AI. When you send a request to the service, you must provide the URI of your JSONL file and your output bucket, including the `gs://` protocols.\n",
      "\n",
      "With the Python SDK, you can create a batch prediction job by calling `Model.batch_predict()`."
     ]
    },
    {
     "cell_type": "code",
     "execution_count": 19,
     "metadata": {
      "id": "f5ab2139d52d"
     },
     "outputs": [
      {
       "name": "stdout",
       "output_type": "stream",
       "text": [
        "INFO:google.cloud.aiplatform.jobs:Creating BatchPredictionJob\n",
        "INFO:google.cloud.aiplatform.jobs:BatchPredictionJob created. Resource name: projects/259224131669/locations/us-central1/batchPredictionJobs/3571004859807170560\n",
        "INFO:google.cloud.aiplatform.jobs:To use this BatchPredictionJob in another session:\n",
        "INFO:google.cloud.aiplatform.jobs:bpj = aiplatform.BatchPredictionJob('projects/259224131669/locations/us-central1/batchPredictionJobs/3571004859807170560')\n",
        "INFO:google.cloud.aiplatform.jobs:View Batch Prediction Job:\n",
        "https://console.cloud.google.com/ai/platform/locations/us-central1/batch-predictions/3571004859807170560?project=259224131669\n",
        "INFO:google.cloud.aiplatform.jobs:BatchPredictionJob projects/259224131669/locations/us-central1/batchPredictionJobs/3571004859807170560 current state:\n",
        "JobState.JOB_STATE_RUNNING\n",
        "INFO:google.cloud.aiplatform.jobs:BatchPredictionJob projects/259224131669/locations/us-central1/batchPredictionJobs/3571004859807170560 current state:\n",
        "JobState.JOB_STATE_RUNNING\n",
        "INFO:google.cloud.aiplatform.jobs:BatchPredictionJob projects/259224131669/locations/us-central1/batchPredictionJobs/3571004859807170560 current state:\n",
        "JobState.JOB_STATE_RUNNING\n",
        "INFO:google.cloud.aiplatform.jobs:BatchPredictionJob projects/259224131669/locations/us-central1/batchPredictionJobs/3571004859807170560 current state:\n",
        "JobState.JOB_STATE_RUNNING\n",
        "INFO:google.cloud.aiplatform.jobs:BatchPredictionJob projects/259224131669/locations/us-central1/batchPredictionJobs/3571004859807170560 current state:\n",
        "JobState.JOB_STATE_RUNNING\n",
        "INFO:google.cloud.aiplatform.jobs:BatchPredictionJob projects/259224131669/locations/us-central1/batchPredictionJobs/3571004859807170560 current state:\n",
        "JobState.JOB_STATE_RUNNING\n",
        "INFO:google.cloud.aiplatform.jobs:BatchPredictionJob projects/259224131669/locations/us-central1/batchPredictionJobs/3571004859807170560 current state:\n",
        "JobState.JOB_STATE_RUNNING\n",
        "INFO:google.cloud.aiplatform.jobs:BatchPredictionJob projects/259224131669/locations/us-central1/batchPredictionJobs/3571004859807170560 current state:\n",
        "JobState.JOB_STATE_SUCCEEDED\n",
        "INFO:google.cloud.aiplatform.jobs:BatchPredictionJob run completed. Resource name: projects/259224131669/locations/us-central1/batchPredictionJobs/3571004859807170560\n"
       ]
      }
     ],
     "source": [
      "job_display_name = \"e2e-text-classification-batch-prediction-job\"\n",
      "model = aiplatform.Model(model_name=model_name)\n",
      "\n",
      "# TODO\n",
      "# create a batch prediction job\n",
      "batch_prediction_job = # TODO: Your code goes here(\n",
      "    job_display_name=job_display_name,\n",
      "    gcs_source=f\"{BUCKET_URI}/{input_file_name}\",\n",
      "    gcs_destination_prefix=f\"{BUCKET_URI}/output\",\n",
      "    sync=True,\n",
      ")\n",
      "\n",
      "batch_prediction_job_name = batch_prediction_job.resource_name"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "id": "11503f2e08a2"
     },
     "source": [
      "Once the batch prediction job completes, the Python SDK prints out the resource name of the batch prediction job in the format `projects/[PROJECT_ID]/locations/[LOCATION]/batchPredictionJobs/[BATCH_PREDICTION_JOB_ID]`. You can query the Vertex AI service for the status of the batch prediction job using its ID.\n",
      "\n",
      "The following code snippet demonstrates how to create an instance of the `BatchPredictionJob` class to review its status. Note that you need the full resource name printed out from the Python SDK for this snippet.\n"
     ]
    },
    {
     "cell_type": "code",
     "execution_count": 20,
     "metadata": {
      "id": "bf6e614723ed"
     },
     "outputs": [
      {
       "name": "stdout",
       "output_type": "stream",
       "text": [
        "Batch prediction job state: JobState.JOB_STATE_SUCCEEDED\n"
       ]
      }
     ],
     "source": [
      "from google.cloud.aiplatform import jobs\n",
      "\n",
      "batch_job = jobs.BatchPredictionJob(batch_prediction_job_name)\n",
      "print(f\"Batch prediction job state: {str(batch_job.state)}\")"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "id": "1f9a12dadf6f"
     },
     "source": [
      "After the batch job has completed, you can view the results of the job in your output Storage bucket. You might want to first list all of the files in your output bucket to find the URI of the output file."
     ]
    },
    {
     "cell_type": "code",
     "execution_count": 21,
     "metadata": {
      "id": "8ff1ec03205c"
     },
     "outputs": [
      {
       "name": "stdout",
       "output_type": "stream",
       "text": [
        "gs://qwiklabs-gcp-00-09d98f4803b0/output/prediction-e2e-text-classification-model-20210824122127-2021-08-24T17:42:17.359307Z/\n"
       ]
      }
     ],
     "source": [
      "BUCKET_OUTPUT = f\"{BUCKET_URI}/output\"\n",
      "\n",
      "! gcloud storage ls --all-versions $BUCKET_OUTPUT"     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "id": "52f3f8af2e41"
     },
     "source": [
      "The output from the batch prediction job should be contained in a folder (or _prefix_) that includes the name of the batch prediction job plus a time stamp for when it was created.\n",
      "\n",
      "For example, if your batch prediction job name is `my-job` and your bucket name is `my-bucket`, the URI of the folder containing your output might look like the following:\n",
      "\n",
      "```\n",
      "gs://my-bucket/output/prediction-my-job-2021-06-04T19:54:25.889262Z/\n",
      "```\n",
      "\n",
      "To read the batch prediction results, you must download the file locally and open the file. The next cell copies all of the files in the `BUCKET_OUTPUT_FOLDER` into a local folder."
     ]
    },
    {
     "cell_type": "code",
     "execution_count": 22,
     "metadata": {
      "id": "4bb16e040942"
     },
     "outputs": [
      {
       "name": "stdout",
       "output_type": "stream",
       "text": [
        "Copying gs://qwiklabs-gcp-00-09d98f4803b0/output/prediction-e2e-text-classification-model-20210824122127-2021-08-24T17:42:17.359307Z/predictions_00001.jsonl...\n",
        "/ [1/1 files][  945.0 B/  945.0 B] 100% Done                                    \n",
        "Operation completed over 1 objects/945.0 B.                                      \n",
        "Local results folder: prediction_results/output/prediction-e2e-text-classification-model-20210824122127-2021-08-24T17:42:17.359307Z\n"
       ]
      }
     ],
     "source": [
      "RESULTS_DIRECTORY = \"prediction_results\"\n",
      "RESULTS_DIRECTORY_FULL = f\"{RESULTS_DIRECTORY}/output\"\n",
      "\n",
      "# Create missing directories\n",
      "os.makedirs(RESULTS_DIRECTORY, exist_ok=True)\n",
      "\n",
      "# Get the Cloud Storage paths for each result\n",
      "! gcloud storage cp --recursive $BUCKET_OUTPUT $RESULTS_DIRECTORY\n",      "\n",
      "# Get most recently modified directory\n",
      "latest_directory = max(\n",
      "    [\n",
      "        os.path.join(RESULTS_DIRECTORY_FULL, d)\n",
      "        for d in os.listdir(RESULTS_DIRECTORY_FULL)\n",
      "    ],\n",
      "    key=os.path.getmtime,\n",
      ")\n",
      "\n",
      "print(f\"Local results folder: {latest_directory}\")"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "id": "f406e1e4d5ec"
     },
     "source": [
      "With all of the results files downloaded locally, you can open them and read the results. In this tutorial, you use the [`jsonlines`](https://jsonlines.readthedocs.io/en/latest/) library to read the output results.\n",
      "\n",
      "The following cell opens up the JSONL output file and then prints the predictions for each instance."
     ]
    },
    {
     "cell_type": "code",
     "execution_count": 23,
     "metadata": {
      "id": "91d7f2a74a7c"
     },
     "outputs": [
      {
       "name": "stdout",
       "output_type": "stream",
       "text": [
        "\n",
        "instance: gs://qwiklabs-gcp-00-09d98f4803b0/input_1.txt\n",
        "\n",
        "ids: ['5078374828348538880', '7384217837562232832', '4213683699893403648', '8825369718320791552', '1619610314527997952', '466688809921150976', '2772531819134844928']\n",
        "\n",
        "displayNames: ['affection', 'enjoy_the_moment', 'achievement', 'nature', 'leisure', 'bonding', 'exercise']\n",
        "\n",
        "confidences: [0.59658015, 0.19601594, 0.18707053, 0.00931904, 0.00615081, 0.0036246842, 0.0012388825]\n",
        "\n",
        "instance: gs://qwiklabs-gcp-00-09d98f4803b0/input_0.txt\n",
        "\n",
        "ids: ['8825369718320791552', '7384217837562232832', '4213683699893403648', '1619610314527997952', '5078374828348538880', '466688809921150976', '2772531819134844928']\n",
        "\n",
        "displayNames: ['nature', 'enjoy_the_moment', 'achievement', 'leisure', 'affection', 'bonding', 'exercise']\n",
        "\n",
        "confidences: [0.3603325, 0.3316431, 0.28240985, 0.019016903, 0.0025975837, 0.0022325595, 0.0017675894]\n"
       ]
      }
     ],
     "source": [
      "# Get downloaded results in directory\n",
      "results_files = []\n",
      "for dirpath, subdirs, files in os.walk(latest_directory):\n",
      "    for file in files:\n",
      "        if file.find(\"predictions\") >= 0:\n",
      "            results_files.append(os.path.join(dirpath, file))\n",
      "\n",
      "\n",
      "# Consolidate all the results into a list\n",
      "results = []\n",
      "for results_file in results_files:\n",
      "    # Open each result\n",
      "    with jsonlines.open(results_file) as reader:\n",
      "        for result in reader.iter(type=dict, skip_invalid=True):\n",
      "            instance = result[\"instance\"]\n",
      "            prediction = result[\"prediction\"]\n",
      "            print(f\"\\ninstance: {instance['content']}\")\n",
      "            for key, output in prediction.items():\n",
      "                print(f\"\\n{key}: {output}\")"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "id": "af3874f08502"
     },
     "source": [
      "## Cleaning up\n",
      "\n",
      "To clean up all Google Cloud resources used in this project, you can [delete the Google Cloud project](https://cloud.google.com/resource-manager/docs/creating-managing-projects#shutting_down_projects) you used for the tutorial.\n",
      "\n",
      "Otherwise, you can delete the individual resources you created in this tutorial:\n",
      "\n",
      "* Dataset\n",
      "* Training job\n",
      "* Model\n",
      "* Endpoint\n",
      "* Batch prediction\n",
      "* Batch prediction bucket"
     ]
    },
    {
     "cell_type": "code",
     "execution_count": 24,
     "metadata": {
      "id": "adce73b48b72"
     },
     "outputs": [
      {
       "name": "stdout",
       "output_type": "stream",
       "text": [
        "INFO:google.cloud.aiplatform.base:Deleting BatchPredictionJob : projects/259224131669/locations/us-central1/batchPredictionJobs/3571004859807170560\n",
        "INFO:google.cloud.aiplatform.base:Delete BatchPredictionJob  backing LRO: projects/259224131669/locations/us-central1/operations/7219005495250518016\n",
        "INFO:google.cloud.aiplatform.base:BatchPredictionJob deleted. . Resource name: projects/259224131669/locations/us-central1/batchPredictionJobs/3571004859807170560\n",
        "INFO:google.cloud.aiplatform.models:Undeploying Endpoint model: projects/259224131669/locations/us-central1/endpoints/7980783159979540480\n",
        "INFO:google.cloud.aiplatform.models:Undeploy Endpoint model backing LRO: projects/259224131669/locations/us-central1/endpoints/7980783159979540480/operations/6552472750399684608\n",
        "INFO:google.cloud.aiplatform.models:Endpoint model undeployed. Resource name: projects/259224131669/locations/us-central1/endpoints/7980783159979540480\n",
        "INFO:google.cloud.aiplatform.base:Deleting Endpoint : projects/259224131669/locations/us-central1/endpoints/7980783159979540480\n",
        "INFO:google.cloud.aiplatform.base:Delete Endpoint  backing LRO: projects/259224131669/locations/us-central1/operations/260944070963101696\n",
        "INFO:google.cloud.aiplatform.base:Endpoint deleted. . Resource name: projects/259224131669/locations/us-central1/endpoints/7980783159979540480\n",
        "INFO:google.cloud.aiplatform.base:Deleting Model : projects/259224131669/locations/us-central1/models/274218199967334400\n",
        "INFO:google.cloud.aiplatform.base:Delete Model  backing LRO: projects/259224131669/locations/us-central1/operations/6016966607207661568\n",
        "INFO:google.cloud.aiplatform.base:Model deleted. . Resource name: projects/259224131669/locations/us-central1/models/274218199967334400\n",
        "INFO:google.cloud.aiplatform.base:Deleting TextDataset : projects/259224131669/locations/us-central1/datasets/7829200088927830016\n",
        "INFO:google.cloud.aiplatform.base:Delete TextDataset  backing LRO: projects/259224131669/locations/us-central1/operations/7851761242896072704\n",
        "INFO:google.cloud.aiplatform.base:TextDataset deleted. . Resource name: projects/259224131669/locations/us-central1/datasets/7829200088927830016\n",
        "INFO:google.cloud.aiplatform.base:Deleting AutoMLTextTrainingJob : projects/259224131669/locations/us-central1/trainingPipelines/1280924449289273344\n",
        "INFO:google.cloud.aiplatform.base:Delete AutoMLTextTrainingJob  backing LRO: projects/259224131669/locations/us-central1/operations/1454397972216283136\n",
        "INFO:google.cloud.aiplatform.base:AutoMLTextTrainingJob deleted. . Resource name: projects/259224131669/locations/us-central1/trainingPipelines/1280924449289273344\n"
       ]
      }
     ],
     "source": [
      "if os.getenv(\"IS_TESTING\"):\n",
      "    ! gcloud storage rm --recursive $BUCKET_URI\n",      "\n",
      "batch_job.delete()\n",
      "\n",
      "# `force` parameter ensures that models are undeployed before deletion\n",
      "endpoint.delete(force=True)\n",
      "\n",
      "model.delete()\n",
      "\n",
      "text_dataset.delete()\n",
      "\n",
      "# Training job\n",
      "job.delete()"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "id": "fa6a8c434c79"
     },
     "source": [
      "## Next Steps\n",
      "\n",
      "After completing this tutorial, see the following documentation pages to learn more about Vertex AI:\n",
      "\n",
      "* [Preparing text training data](https://cloud.google.com/vertex-ai/docs/datasets/prepare-text)\n",
      "* [Training an AutoML model using the API](https://cloud.google.com/vertex-ai/docs/training/automl-api#text)\n",
      "* [Evaluating AutoML models](https://cloud.google.com/vertex-ai/docs/training/evaluating-automl-models#text)\n",
      "* [Deploying a model using the Vertex AI API](https://cloud.google.com/vertex-ai/docs/predictions/deploy-model-api#aiplatform_create_endpoint_sample-python)\n",
      "* [Getting online predictions from AutoML models](https://cloud.google.com/vertex-ai/docs/predictions/deploy-model-api#aiplatform_create_endpoint_sample-python)\n",
      "* [Getting batch predictions](https://cloud.google.com/vertex-ai/docs/predictions/batch-predictions#text)"
      ]
    }
  ]
}
